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discussed  findings  of  ARI  sponsored  National  Research  Council  reports  on  value 
of  "non-mainstream"  training  techniques  for  US  Army  training.  Training 
Technology  demonstrations  at  Williams  Air  Force  Base,  Armstrong  Laboratory 
included  night  vision  devices,  MULTIRAD  (simulator)  and  visual  color  modeling. 
During  the  Steering  Committee  meeting  it  was  decided  that  we  should  encourage 
our  respective  laboratories  to  view  T2TG  as  a  mechanism  for  implementing 
TAPSTEM.  Changes  in  the  chairmanships  of  subgroups  will  take  place  next  year: 
the  Navy  taking  over  Advanced  Technologies,  and  the  Army  assuming  chairmanship 
of  the  Crew,  Group  and  Unit  subgroups.  In  addition,  the  chair  of  the  Steering 
Committee  will  rotate  from  the  Army  to  the  Navy  following  next  year's  meeting. 
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DOD  Training  Technology  Technical  Group 
(T2TG)  Minutes 


The  sixth  meeting  of  the  DOD  Training  Technology  Technical 
Group  (T2TG)  was  held  on  24-25  March  1992,  at  Phoenix 
Arizona.  It  was  hosted  by  Armstrong  Laboratory,  Williams  Air 
Force  Base. 

There  were  three  keynote  presenters.  Dr.  William  Howell, 
AL/HRD  spoke  about  situational  awareness  as  it  relates  to  aircrew 
performance.  Mr.  Denis  Breglia,  NTSC,  described  the  issues 
surrounding  the  development  and  use  of  virtual  environments  as 
training  technologies.  Dr.  Michael  Drillings,  of  the  Basic  Research 
Office,  USARI,  discussed  the  findings  of  ARI  sponsored  National 
Research  Council  reports  on  the  value  of  "non-mainstream" 
training  techniques  for  U.S  Army  training.  Training  Technology 
demonstrations  at  Williams  Air  Force  Base,  Armstrong  Laboratory 
included  night  vision  devices,  MULTIRAD  (simulator),  and  visual 
color  modeling. 

During  the  Steering  Committee  meeting,  it  was  decided  that  we 
should  encourage  our  respective  laboratories  to  view  T2TG  as  a 
mechanism  for  implementing  TAPSTEM.  Changes  in  the 
chairmanships  of  subgroups  will  take  place  next  year:  the  Navy 
taking  over  Advanced  Technologies,  and  the  Army  assuming 
chairmanship  of  the  Crew,  Group,  and  Unit  subgroups.  In  addition, 
the  chair  of  the  Steering  Committee  will  rotate  from  the  Army  to 
the  Navy  following  next  year’s  meeting. 

The  next  meeting  is  scheduled  for  4-5  May  1993.  It  will  be  hosted 
by  the  Army  Research  Institute  in  Orlando,  Florida. 


The  following  pages  provide  the  agenda,  subgroup  summaries, 
hardcopies  of  the  viewgraphs,  and  the  list  of  attendees. 
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AGENDA 


6th  DoD  TRAINING  TECHNOLOGY  TECHNICAL  GROUP 
24-25  March  1992 


TUESDAY.  24  MARCH  1992 

0700  -  0800  Registration/Fees 
PLENARY  8E88ION  I 
0800  -  0810 


0810 

0830 

0845  -  0900 

0900  -  0940 

0940  -  1020 


Commander's  Welcome 
(Lt  Col  Lynn  Carroll) 

Aircrew  Training  Research  Division  Overview 

Administrative  Support  Announcements 
(Ms  Linda  Swan) 

Introduction  of  Invited  Speakers 
(Dr  Bob  Seidel,  ARI) 

Situational  Awareness 

(Dr  William  Howell,  HR  Directorate  Armstrong  Laboratory) 

Enhancing  Human  Performance 
(Dr  Michael  Drillings,  ARI) 


1020  -  1050  Coffee  Break 

1050  -  1130  Virtual  Environments 

(Mr.  Denis  Breglia,  NTSC) 


1130  -  1300  LUNCH 

SUBGROUP  SESSION  I 


Advanced  Training  Technology 

Introduction  and  Administrative  Issues  Subgroup  Theme: 
Simulator,  Simulations  and  "Virtual  Reality" 

Enhancing  Aircrew  Training  Through  Virtual  Environment 
Research  (Dr.  Richard  Thurman,  USAF  AL/HRAU) 

Research  on  the  Use  of  Virtual  Environments  in  Crisis 
Management  in  the  Navy  (Ms.  Janet  Dickieson,  NPRDC) 

Behavioral  Requirements  for  Training  in  Virtual 
Environments  (Dr  Bruce  Knerr,  USARI) 
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Crew*  Group#  Team,  and  Unit  Technology  Sub-Group 
Opening  Eeaarlcs 


Joint  Collective  Training  R&D  Effort 

-  Dr  Frank  Moses,  ARI 

-  Dr  Eduardo  Salas,  NTSC 

-  Discussion  (All) 

Training  Design  &  Evaluation 

Welcome  and  Administrative  Issues 

Training  Needs  and  Evaluation  Issues 

-  Identifying  Over-and-Under-Trained  Tasks  (Ms  Morales) 

-  Opportunities  to  Perform  Trained  Tasks 
(Dr  Mark  Teachout) 

Roundtable  Discussion  (ALL) 

1600  Adjourn  from  Subgroup  Location 

1700  -  1900  No  Host  Bar  with  Heavy  Hors  d' Oeuvres  -  Resort's  Lounge 
WEDNESDAY.  25  MARCH  1992 
SUBGROUP  SESSION  II 

Advanced  Training  Technology 

Visual  Learning  in  Virtual  Environment  (Dr  J.  Psotka,  ARI) 

Summary  and  Conclusions  of  Virtual  Reality  in  Training 
Research  in  the  Services  or  "What  are  the  Research  Issues 
in  the  use  of  Virtual  Reality  in  Training?" 

Roundtable  Discussion 

Crew.  Group  and  Unit  Training 

Aircrew  Coordination  Training  R&D 

-  Dr  David  Baker,  NTSC 

-  Mr  Randall  Oser,  NTSC 

-  Major  Wes  Woodruff,  USAF,  NTSC 

-  Discussion  (All) 

Training  Design  and  Evaluation 

Instructional,  Planning  and  Evaluation  Issues 

-  Modeling  Skill  Acquisition  (Dr  Sabol) 

-  Retention  of  Knowledge  Learned  in  College  (Dr  Ellis) 

Roundtable  Discussion  (ALL) 
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0945  -  1000  BREAK 


SUBGROUP  SESSION  III 

Advanced  Training  Technology 

Basic  Job  Skills  Job  Family  Tutor 
(Dr  Ellen  Hall,  US AF-AL/HRMJ C ) 

Issues  in  Designing  and  Intelligent,  NLP-based  Tutor  for 
Foreign  Languages  (Dr  Michelle  Sams,  USARI) 

Summary  and  Conclusions 

Crew.  Group  and  Unit  Training 

Update  of  AF  ISD  Process 

-  Major  Conrad  Bills,  ASD/YTEE 

Team  Decision-Making  Training  (Update) 

-  Eduardo  Salas 

Discussion  (All) 

-  Next  Meeting 

-  Topics 

-  Format 

-  Product (s) 

Treihing.Bssiqb  Evaluation 

Instructional,  Planning  and  Evaluation  Issues 

-  Instructional  Strategies  for  Logistic  Command  and 
Control  (Captain  Hioki) 

-  Distance  Learning  (Mr  Gettman) 

Roundtable  Discussion  (ALL) 

1145  -  1245  LUNCH 

1245  Bus  departs  Conference  Center 

1300  Arrival  Williams  Air  Force  Base,  Armstrong  Laboratory 

1300  -  1500  DEMONSTRATIONS 


3 


flBQgg  1 


GROUP  2 


1305  -  1325 

Night  Vision  Devices  &  Training 
(Bldg  558) 

1435  -  1500 

1335  -  1400 

MULTIRAD 
(Bldg  561) 

1405  -  1425 

1435  -  1500 

Visual  Systems  Color  Modeling 
(Bldg  558) 

1305  -  1325 

1510 

Bus  departs  Williams  Air  Force  Base 

1525 

Arrival  Conference  Center 

1530  -  1630 

Wrap-Up 
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DR.  WILLIAM  HOWELL 

HR  Directorate  Armstrong  Laboratory 
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SITUATIONAL  AWARENESS 


\ 


DR  WILLIAM  C.  HOWELL 
HUMAN  RESOURCES  DIRECTORATE 
ARMSTRONG  LABORATORY 


OVERVIEW 
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Table  1 

Characteristics  of  Pilot  Situation  Awareness 


External 

Situation 

Benefits  of 

Msslon 

Information 

Situation 

Components 

Awareness 

Categories 

Sources 

Novice 

Expert 

Routine 

Spatial 

Mld-elr 

Locsl  navi- 

Sensory  In- 

Expending 

Utilizing 

orientation 

collision 

gsllon, 

formation 

unnecessary 

non-competing 

avoidance 

guidance 

from  the 

effort 

r*iourc«s 

Environment 

and  control 

environment 

Routine 

Terrain 

Not 

Multiplexing 

avoidance 

Communl- 

Cockpit  visual 

perceiving 

goals 

Robust 

cation  outside 

and  auditory 

patterns 

Shortening 

the  cockp't 

displays 

transmissions 

Procedures 

decision 

tor  attaining 

making  In 

Flight  crew 

Extra-  and 

Converting 

goals 

the  face  of: 

resource 

Intra-aircraft 

Interference 

management 

eommunlca- 

Aircraft 

Turbulence 

tlon 

Chunking 

ay  stem 

Cross  winds 

Cabin 

status 

Wlndshesr 
Loss  of 

management 

Recorded  flight 
plans 

Aircraft  par- 

visibility 

Routine 

forma  nee 

Craw  respon- 

management 

of  physical 

Flight  manage¬ 
ment  computer 

equipment, 

aJbilldes  A 

resources, 

Flight  manuals 

knowledge 

and  systems 

and  checklists 

Routine 

Future  3D 

management 

navigation 

of  FMC  end 
related  crew 
elding  systems 

elds 

Bridging 

activities 

1  Non-Routine  Soedal 

Improved 

Macro -planning 

Future  route 

Making 

Shedding, 

delaying. 

constraints 

decision 

A  navigation 

diversion 

last 

making 

aids 

minute 

and 

Contingency 

plans 

In  the 
lace  of: 

plans 

during 

pre-loading 

tasks 

Oo  around 
Weather 

high 

workload 

rerouting 

Emeroencv 

Unusual 

Improved 

Diagnosis  of 

Future  fsult 

Fixating  on 
one  or  two 

Letting  go  of 
high  workload 

symptoms 

fault 

physical 

finding 

management 

equipment, 

aids 

salient 

strategies 

Trouble- 

rtMurcM, 

possibilities 

shooting 

techniques 

and  systems 

Diagnosis  of 

Ignoring  vital 
flight 

Emergency 

FMC  A  related 

Information 

procedures 

crew -elding 

while 

systems 

trouble¬ 

shooting 

Hof:  Tho  rows  srt  cumula  live.  In  othsr  words,  the  entries  for  emsrgency  situations  Include  all  the  entries 
tor  routine  and  non-routine  equations. 


11 


RRENT  PERSPECTIVES 


THEORETICAL  ISSUES 
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THEORETICAL  ISSUES  (CONT.) 
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QUALITY  OF  MEASURES  (PSYCHOMETRIC) 


Tentative  Conclusions 
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RESEARCH  STRATEGIES 
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3.1.1  Dr  Grant  McMillan  (AL/CFIIP).  Serves  as  project  leader  and  responsible 
for  overall  execution  of  the  study. 

3.1.2  Lt  Col  Jim  Bushman  (AL/CCE).  Assists  Dr  McMillan  In  management  and 
execution  of  the  study. 

3.1.3  Maj  David  Perry  (AL/I1RMAA).  Responsible  for  development,  test,  and 
evaluation  of  the  SAAB.  Responsible  for  overall  analyses  of  the  study. 

3.1.4  Dr  Wayne  Waag  (AL/HRAT).  Responsible  for  development,  test,  and 
evaluation  of  the  SA  Rating  Scale.  Assists  in  overall  study  analyses. 

3.1.5  Dr  Mike  Vidullch  (AL/CFIIP).  Responsible  for  Identification  of  the 
cognitive  and  performance  components  of  SA.  Advises  on  the  development  of  the 
SAAB  and  the  SA  Rating  Scale. 

3.1.6  Dr  Sam  Schiflett  (AL/CFTO).  Advises  on  the  development  of  the  SAAB  and 
SA  Rating  Scale. 

3.1.7  Maj  Glen  Larsen  (USAFSAM/FP) .  Consultant  to  overall  study. 

3.1.8  Lt  Col  Tim  Kinney  (WL/F1P).  Joint  Cockpit  Office  representative. 

Serves  as  consultant  to  overall  study. 
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SCENARIOS  AND 


Develop  Computerized  SA  Test  Battery 
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Develop  Behaviorally  Anchored  SA  Rating  Scale 


DR.  MICHAEL  DRILLINGS 

U.S.  Army  Basic  Research  Office 
5001  Eisenhower  Ave,  Alexandria,  VA 
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ENHANCING  HUMAN  PERFORMANCE 
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Recommend  Research  to  Better  Understand  Performance 
Enhancement 
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Cost  —  Benefit  Factors 
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Biofeedback  Not  Proven 
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Enhancing  Aircrew  Training  Through  Virtual 
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Dr.  Richard  Thurman 

Research  on  the  Use  of  Virtual  Environment  in 
Crisis  Management  in  the  Navy: 

Ms.  Janet  Dickieson 
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Spatial  awareness  consists  of  a  description  of  each  object's 
3  coordinates  of  location,  3  coordinates  of  orientation  and 
a  motion  velocity  vector. 
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The  air-to-air  intercept  as  a  spatial  awareness  problem. 
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ARI  OPTION  TO  NTSC  VETT  CONTRACT 
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ARI  OPTION  TO  NTSC  VETT  CONTRACT 
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MAKING  VIRTUAL  ENVIRONMENTS  A  REALITY 
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AN  EXAMPLE 
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PRELIMINARY  VIRTUAL  ENVIRONMENT  INTERFACE  ISSUES 
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HUMAN  FACTORS  ISSUES 


BEHAVIORAL  REQUIREMENTS  FOR  TRAINING  AND  REHEARSAL  IN  VIRTUAL 
ENVIRONMENTS  (ILLUSION  ENGINEERING,  INC) _ 
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IDENTIFY  REQUIREMENTS  &  COST  FOR  RESEARCH  TEST  BED 


VIRTUAL  ENVIRONMENTS 
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CREW.  GROUP .  TEAM.  AND  UNIT  TECHNOLOGY  SUB-GROUP 

Opening  Remarks 

Joint  Collective  Training  R&D  Effort: 

Dr.  Frank  Moses,  ARI 

Dr.  Eduardo  Salas,  NTSC 
(No  hard  copies  available) 
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ASSESSMENT  OF  JOINT  TRAINING  STRATEGIES  FOR 
INCREASING  WARFIGHTING  EFFECTIVENESS 
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Dr.  Frank  Moses 
POC  @  ARI 
Tel:  274-8293) 


ASSESSMENT  OF  JOINT  TRAINING  STRATEGIES 
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ENHANCED  TRAINING  STRATEGIES 
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Commanders  Conference 
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★★★★  Commanders  Conference 


SYNTHETIC  BATTLEFIELDS  ON  DEMAND 


SIMULATION  TECHNOLOGY  FOR  THE  SYNTHETIC  BATTLEFIELD 
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SYNTHETIC  BATTLEFIELDS  ON  DEMAND 
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CENTCOM 

SIM 

TEST  BED 


DRAINING  DESIGN  &  EVALUATION 


Training  Needs  and  Evaluation  Issues 

Identifying  Over-and-Under  Trained  Tasks 
Ms.  Michele  Morales 

Opportunities  to  Perform  Trained  Tasks 
Dr.  Mark  Teachout 
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AN  INVESTIGATION  OF 
TRAINING  EFFICIENCY 


TRAINING  EFFICIENCY  RESEARCH 
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STEP  1 

IDENTIFICATION  OF  TRAINING  CONTENT  DOMAIN 
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AS  CONSISTING  OF  99  TASKS 


STEP  2 

IDENTIFICATION  OF  TRAINING  EMPHASIS 
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99  TASKS  TRAINED 


STEP  3 

APPLICATION  OF  THE  MATCHING  TECHNIQUE 
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OR  UNDERTRAINED  TASKS 


MATCHING  TECHNIQUE  TO  EXAMINE  TRAINING  EFFICIENCY 
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ACTUAL  TRAINING  EMPHASIS 


TASK  441  ADJUST  PNEUMATIC  SYSTEM  CLUTCHES 

H 

OSR  Emphasis  M 

L 

Figure  la:  Possible  overtrained  task 
TASK  246  ADJUST  GAS  TURBINE  ENGINE  GOVERNORS 

H 

OSR  Emphasis  M 

L 

Figure  1b:  Possible  undertrained  task 
FIGURE  1.  Examples  of  Possible  Over  and  Undertrained  Tasks. 


L  M  H 
Instructor  Emphasis 


TASK  264:  ISOLATE  ENGINE  OR  MOTOR  MECH.  MALF. 

H 

OSR  Emphasis 

M 

L 

Figure  2a:  Training  hit,  high  emphasis 
TASK  226:  REMOVE  OR  INSTALL  CANNON  PLUG  PARTS 

H 

OSR  Emphasis 

M 

L 

Figure  2b:  Training  hit,  low  emphasis 
FIGURE  2.  Examples  of  Tasks  that  are  Training  Hits. 


L  M  H 


Instructor  Emphasis 


STEP  4 

LINKING  MATCHING  OUTCOMES  &  PERFORMANCE 
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JOB  PERFORMANCE  LEVEL 
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FIGURE  4.  Results  of  Linking  Matching  Outcomes  to  Performance. 


CURRENT  RESEARCH 
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Cluster  by  equipment 

Cluster  by  function 

Rank  order  by  Z  performing  first  term 

Rank-order  by  difficulty  rating 

Rank  order  by  Z  performing  1st  12  mo. 

Rank  order  by  ATI 

Return  to  main  menu 


Use  the  UP  and  DOWN  ARROWS  to  highlight  an  option 
Press  RETURN  to  select  the  highlighted  option 


Screen  11 
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Task:  381  Isolate  electrical  circuitry  malfunctions 


OLD  OSR 
EMPHASIS 


NEW  OSR 
EMPHASIS 


Press  any  key  to  continue 


Screen  17 
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PART  I:  For  each  task  statement  listed  below,  answer  the  following  questions  in  the  appropriate  column.  When  completing  these 
questions,  only  consider  the  first  twelve  months  since  graduation  from  Chanute  AFB  regardless  of  how  long  you  have  been  at  your 
present  duty  station. 

1  In  the  first  1 2  months  since  completion  of  training  at  Chanute  AFB,  have  you  PERFORMED  this  task  either  with  or  without 
supervision?  Completely  fill  in  the  circle  *Y*  if  you  have  performed  the  task  or  fill  in  the  circle  ‘N*  if  you  have  not  performed  the 
task. 

For  every  item  that  you  answer  yes,  please  answer  the  following  two  questions: 

2.  WHEN  was  the  iiQl  time  you  performed  the  task?  Fill  in  *Y*  if  you  first  performed  the 

task  in  the  first  eight  months  after  graduation.  Otherwise,  fill  in  the  number  _____ 

corresponding  to  the  month  in  which  you  first  performed  the  task. 

3.  HOW  MANY  TIMES  have  vou  performed  the  task  since  graduating  from  Chanute 

AFB?  Use  the  boxes  to  mark  your  answer.  For  example,  if  you  have  performed  "  i  >  -- 

'Adjust  contactor  points*  two  times  in  the  first  twelve  months  since  graduating  from 

Chanute,  you  would  write  in  the  boxes  QQQ  I 


PERFORMED? 

WHEN? 

HOW  MANY 

YES  NO 

(MONTH) 

OF  TIMES? 

1. 

Fill  out  AFTO  Forms  244 . 

©® 

© 

©0©0 

□  □□ 

2. 

Make  entries  on  AFTO  Forms  350  (item  processing  tag) . 

©0 

© 

0©®0 

□  □□ 

3 

Perform  generator  inspections . 

...  0® 

© 

©00® 

□  □□ 

4. 

Measure  resistance  in  electrical  circuits . 

. ©0 

© 

©00© 

□  □□ 

5 

Perform  electrical  system  operational  checks . 

. o® 

© 

©00© 

□  □□ 

6 

Solder  electrical  wiring . 

. ©0 

© 

©00© 

□  □□ 

7. 

Splice  electrical  wiring . 

. ©0 

© 

©00© 

□  □□ 

8. 

Adjust  engine  fuel  system  components . 

. ©0 

© 

©00© 

□  □□ 

9. 

Isolate  engine,  motor,  or  generator  mechanical  malfunctions . 

. ©0 

© 

©@0© 

□  □□ 

10. 

Perform  compression  tests . 

. ©0 

© 

©@©@ 

□  □□ 

11. 

Perform  engine,  motor,  or  generator  operational  checks . 

. ©0 

© 

©00© 

□  □□ 

12. 

Remove  or  install  AGE  tire,  tube,  or  wheel  assemblies . 

. ©0 

© 

©00© 

□  □□ 

13. 

Perform  air  conditioner  visual  or  service  inspections . 

. ©0 

© 

©00© 

□  □□ 

14. 

Perform  hydraulic  test  stand  service  inspections . 

. ©0 

© 

©00© 

□  □□ 

15. 

Perform  hydraulic  test  stand  periodic  inspections . 

. ©0 

© 

©0  0  @ 

□  □□ 

16. 

Clean  motor  or  generator  armatures . 

. ©0 

© 

©00© 

□  □□ 

17. 

Charge  ref rigerant  systems . 

. ©0 

© 

©00© 

□  □□ 

16 

Perform  air  conditioner  leakage  tests . 

. ©0 

© 

©00© 

□  □□ 

19. 

Perform  air  conditioner  operational  checks . 

. ©0 

© 

©00© 

□  □□ 

20. 

Purge  refrigerant  systems . 

. ©0 

© 

©00© 

□  □□ 

21. 

Adjust  hydraulic  high  pressure  system  components . 

. ©0 

© 

©00© 

□  □□ 

22 

Isolate  hydraulic  system  malfunctions . 

. ©0 

© 

©00© 

□  □□ 

23 

Perform  hydraulic  system  operational  checks . 

. ©0 

© 

©00© 

□  □□ 

24. 

Pack  wheel  bearings . 

. ©0 

© 

©00© 

□  □□ 

25. 

Perform  load  bank  inspections . 

. ©0 

© 

©00® 

□  □□ 

26. 

Remove  or  install  engine  fan  belts . 

. 0® 

© 

©00© 

□  □□ 

27. 

Perform  generator  periodic  inspections . 

. ©0 

© 

©Q00 

□  □□ 

28. 

Perform  gas  turbine  compressor  periodic  inspections . . 

. ©® 

© 

©00© 

□  □□ 

29. 

Adjust  turbine  engine  bleed  air  systems . 

. ©0 

© 

©00© 

□  □□ 

30. 

Adjust  turbine  engine  fuel  systems . 

. ©0 

© 

©00© 

□  □□ 

31. 

Load  test  generator  sets . 

. ©® 

© 

©00© 

□  □□ 

32. 

Perform  air  compressor  periodic  inspections . 

. 0® 

© 

©©  ®@ 

□  □□ 

33. 

Isolate  heating  system  malfunctions . 

. ©0 

© 

©00© 

□  □□ 

34 

Perform  heating  system  operational  checks . 

. 0® 

© 

©00© 

□  □□ 

35. 

Isolate  air  compressor  system  malfunctions . 

. ©0 

© 

©00© 

□  □□ 

36 

Remove  or  install  air  compressor  filtering  system  components . 

. .-.  ©® 

© 

©00© 

□  □□ 

37. 

Remove  or  install  fuel  lines  or  fittings  other  than  diesel . 

. ©® 

© 

©00© 

□  □□ 

38 

Remove  or  install  hydraulic  lines  or  fittings . 

.  ©0 

© 

©00© 

□  □□ 

39 

Research  TO's.  charts,  or  diagrams  for  AGE  enclosures,  chassis,  or  drives 

. 0® 

© 

©00© 

□  □□ 

to 

Inspect  vehicles  for  safety  of  operation . 

. ©® 

© 

©0  o© 

□  □□ 
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A1 


%  PERFORMING  TASK  BY  MONTH 
After  4  Months 


MONTH 


OPPORTUNITY  TO  PERFORM 
CONSTRUCT 


DIMENSION 

DEFINITION 

Breadth 

#  of  Tasks 

Activity  Level 

#  Times 

Task  Type 

Difficulty/Criticality 

165 


METHOD 


TRAINING  COURSE 

Aerospace  Ground  Equipment 
1 8  Weeks 
99  Tasks  Taught 

SAMPLE 

1 80  Recent  Graduates 
34  Tasks  Sampled 

COLLECTION 
Survey  Methodology 

Measures  gathered  4  months  after  training 
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SUPERVISOR 

Attitudes  Toward  Airman 
Work  Flow 


MEASURES 


AIRMAN 

Opportunity  to  Perform 

Support 

Seif  Efficacy 


OTHER 

MAJCOM 
Ability  (ASVAB) 
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Hierarchical  Block  Regression  Results 

For  Breadth 


STEP 

VARIABLES 

R2 

AR2 

1 

MAJCOM 

.04 

.04 

2 

Sup  Attitudes3 
Support 

Work  Flow 

.10* 

.06* 

3 

Self  Efficacy  * 
Ability  (AFQT) 

.20* 

.10* 

NOTE:  *  (X.05 

a 

Beta  for  this  variable  was  significant  (P<.05)  at  this  step 
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Hierarchical  Block  Regression  Results 
For  Activity  Level 


STEP 

VARIABLES 

R2 

AR2 

1 

MAJCOM  a 

.07* 

.07  * 

2 

Sup  Attitudes 
Support 

Work  Flow 

.09* 

.02 

3 

Self  Efficacy 
Ability  (AFQT)* 

.13* 

.04* 

NOTE:  ‘  pc.05 
a 


Beta  for  this  variable  was  significant  (P<.05)  at  this  step 
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Hierarchical  Block  Regression  Results 
For  Task  Type 


STEP 

VARIABLES 

R2 

AR2 

1 

• 

MAJCOM 

.03 

.03 

2 

Sup  Attitudes3 
_  a 

Support 

Work  Flow 

.50* 

.47  * 

3 

Self  Efficacy  8 
Ability  (AFQT) 

.55* 

.05* 

NOTE:  *  (X.05 

a 

Beta  for  this  variable  was  significant  (P<.05)  at  this  step 


Findings 


C  T3 
—  o 

mS 

©  CQ 

o 

c 

£  E 

©  i_ 

^  5 

© 

©  a 

2S 

£  © 
■o  © 
c  ~ 


0  k- 
o 

2  a 
0  o 


172 


These  differences  are  predictable 


Visual  Learning  in  Virtual  Environment: 

Dr .  J .  Psotka 

Summary  and  Conclusions  of  Virtual  Reality  in  Training 
Research  in  the  Services  or  "What  are  the  Research 
Issues  in  the  use  of  Virtual  Reality  in  Training?" 


Visual  Communication  in  Multi-Media  Virtual  Realities. 


This  basic  research  project  in  visual  communication  examines  how  visual  knowledge 
should  be  structured  to  take  full  advantage  of  advanced  computer  environments  for  training, 
especially  hypertext  and  virtual  reality.  A  Visual  AirCraft  Recognition  ( VAC R)T raining 
hypertext  has  been  built  and  tested.  Virtual  Reality  workstations  have  been  explored  and  will  be 
used  for  future  experiments.  A  theory  of  visual  concept  learning  is  under  development. 

6.1  RESEARCH 

Our  first  experiments  examined  the  interface,  architecture,  and  training  strategy  issues  for 
combining  images  and  text  in  multimedia  systems.  An  Army  Field  Manual  (FM  44-30)  was 
completely  digitized,  and  redesigned  to  take  advantage  of  several  powerful  hypertext  features: 
search,  browsing,  “hot"  words  or  buttons,  apparent  motion,  and  colored  contrasts.  A  series  of 
experiments  determined  the  training  advantages  of  these  features.  These  results  were 
integrated  into  a  theoretical  framework  that  combined  “ecological  perception”  with  “apparent 
motion”  as  a  basis  for  visual  concept  learning.  The  theory  is  continuing  to  be  refined  and  tested 
in  ongoing  work. 

The  hypertext  and  digitized  images  are  being  transferred  from  our  experimental  multimedia 
platform  to  a  state-of-the-art  Virtual  Reality  Platform  for  research.  This  work  will  examine 
the  value  of  virtual  “immersion"  into  a  3-D  environment  for  memory  of  spatial  orientation, 
over  2-D  spatial  interfaces.  In  addition,  the  Virtual  Reality  workstation  will  allow  us  to  extend 
our  theoretical  framework  to  begin  to  analyze  the  comparative  strengths  of  speech 
communication  versus  visual  communication  in  the  exchange  of  shared  mental  models  among 
crew  and  group  members.  Interactions  between  people  and  simulated  crew  members  will  use 
detailed  models  of  animated  agents  and  faces  developed  at  the  Army  Center  for  Al  at  the 
University  of  Pennsylvania. 

THEORY 

Basic  theories  about  visual  communication  need  to  be  developed  in  detail  if  the  rapid  progress 
in  computer  technologies  is  to  be  fully  leveraged  in  future  Army  training.  Recent  synthetic 
reality  and  hypermedia  computer  technologies,  combined  with  Artificial  Intelligence  (Al) 
knowledge  representation  techniques,  offer  unprecedented  opportunities  for  digitizing, 
displaying,  transforming,  and  transmitting  pictures  as  easily  as  words  and  sentences. 

POTENTIAL  MILITARY  RELEVANCE 

The  Army  is  increasingly  turning  to  large  scale  simulator  networking  for  cost  effective 
training  of  warfighting  skills.  This  research  advances  several  core  technologies  that  will  be 
transferred  to  6.2  research  in  AISTA,  PM-TRADE  Field  Unit,  and  Ft.  Bliss  Field  Unit.  The 
v  ACR  training  hypertext  is  being  transferred  to  Ft.  Bliss  currently.  Future  work  will  have 
direct  bearing  on  distributed  simulator  design  for  “popped  hatch”  tank  simulators,  and 
effective  crew  communication  and  spatial  navigation  training. 

ARI  POC:  Dr.  Joseph  Psotka,  AV  284-5540;  Comm  (703)  274-5540. 
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anal  ©®ifa©@®$  TraHsaSra® 


o  Digitize  a  multimedia  HyperBook  for  training 

o  Conduct  experiments  to  determine  principles  of  training 
design  in  multimedia  environments 


FT  Bliss  IPR,  Mar  23,  1992 
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Protect  SHORAD  Lethality 
VACR  -  Visual  Aircraft  Recognitior 


Bliss  IPR,  Mar  23,  1992 


m  AflFtsiraffc 


Basic  Research  Briefing 


FT  Bliss  IPR,  Mar  23,  1992 
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FIRST  EXPERIMENT 
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SECOND  EXPERIMENT 
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Main-Planes  3/18/92  9:19  AM 


1 


Aircraft  Name 


DIRECTIONS: 


Click  on  the  name 
of  any  plane  to  go 
to  the  card  with 
information  about 
that  plane. 


JAGUAR 
AV-8  HARRIER 
KING  AIR  U-21 
HAWK 
DRAKEN 
F-15  EAGLE 

AN-24  COKE,  AN-26  CURL 

YAK-36  FORGER 

U-8F  SEMINOLE, QUEEN  AIR 

TU-26  BACKFIRE 

C-141B  STARLIFTER 

HUNTER 

A-6  INTRUDER 

F-4  PHANTOM 

MIG-25  FOXBAT 

MIRAGE-lll/5 

C-5A  GALAXY 

B-1B 

An-32  CLINE 
KRR  C-2 
TORNADO 


Main  Menu 
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o  MILLARD  FILLMORE  Q 
O  JOHN  ADAMS  Q 

O TORNADO  O 

O  GEORGE  MRSON  O 


ABP  VBPA 


Similarity  Space  of  Planes 


I- 

2 

LU 

2 

cc 

LU 

CL 

X 


.as 

s 

t 


o 

o 

CO 

CC 

LU 

Q. 


00 

3 

GO 

CC 

LU 


3 

O 


CO 

c 


C/5 

O 

c 

jo 

CL 

O 

CM 

O 

CO 

o 

E 

CO 

c 


o 

CQ 

I 


co 

o 


UL 


CO  t 
* 

h-  CO 
0)  " 
W  * 

c 

CO 


(O 

05 

K 

+-» 

CO 

o 

CL 


* 

o 

LO 

uo 


* 

+ 

o 

a 

O 

CO 


* 

CNJ 

CO 

M" 


* 


CO 

CO 


J*  n 


05 

o 

00 


in  o> 
^  'Jo 
V  -M 

^  o 


♦ 

*  + 


189 


Transfer-B 


06 1 


Rank  Ordered  Groups 


Percent  Correct  Recognition 


191 


Number  of  Views 


Number  of  Similar  Planes  viewed  by  Both  Groups 


AAnAn,*AVb1c14cdrakf15FH*HuJagiKf KiMiMiriTuSlYak  36 

Plane 


192 


Number  of  Views 


Number  of  Looks  at  Each  Plane 


Pictures  Viewed 


Transfer  Test 
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Data  from  "Hyper  Group  -  Subject  Measures 


Hyper  Group  Subjects 


Number  of  Contrasts  Viewed 
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Data  from  NHyper  Group  -  Subject  Measures 


Flash  +  Cycle 


196 


9 


3/21/91  9:56  AM  WEFT  Matrix  1 


W.E.F.T.  Tm%  Earn 

V I  ml  _ 


Cockpit 


Test  yourself  on  any  WEFT  feature  by  buttoning  that  feature  and 


then  answerlnq  all  the  questions  that  come  up  ... 


Attack 


Ground  Attack 


Interceptor 


Fighter 


Strike 


Fighter-bomber 


Bomber 


Cargo 


Transport 


Aeconnaissance 


Surueiliance 


Obseruation 


Trainer 


Main  Menu 


Click  on  any  button  to  see  all  the  aircraft  that  have 

the  selected  Role. 


Jet,  swept  wing, 
fuselage  engine(s) 


Jet  Aircraft _ 


Jet,  swept  wing, 
rear  engines 


Guided  Tour _ 

To  get  an  overview  of  the  different  aircraft  listed  below,  click  the 
"Take  Tour"  button.  On  the  tour,  click  anywhere  to  go  to  the  next 


USA 


AWACS 
A-6  INTRUDER 
A-10  Thunderbolt  II 
B-52 

AH-64  APACHE 
F-4  PHANTOM 
F-15  Eagle 
F-16  FIGHTING 
FALCON 
F-1 1 1 
F-117A 


Take  Tour _ | 

Main  Menu) 


I  IRAQ  t 


MIRAGE-F1  | 

Mi-24 

HIND 

MiG-23 

Flogger  B 

MiG-29 

Fulcrum 

Su-24 

FENCER 

Take  Tour 


A-10  THUNDERBOLT  II 


]  Country  o  1  Origin.  USA. 

Similar  Aircraft.  None. 

Crew.  One. 

Role.  Close  air  support,  ground  attack. 


Armament.  30-mm  cannon,  bombs,  rockets,  HELLFIRE 
missiles,  gun  pods. 

Dimensions.  Length  53  feet,  span  58  feet. 

Wings.  Wings  are  low  mounted  on  the  fuselage, 
unequally  tapered,  with  blunt  curled-under  tips. 

Landing  gear  pods  extend  forward  of  the  wings'  leading 

•does.  I  Sound  f 

Engine(s).  Two  turbofan  engines  In  pods,  high  on  the 
rear  of  the  body  between  the  wings  and  the  tall  section. 


Fuselage.  Rounded  nose,  tapered  rear,  bubble  canopy. 
Protrusion  In  nose  Is  the  30-mm  GAU-8  cannon. 


Tail.  Two  tall  fins  on  tips  of  flat  Unequally  tapered 
fins  extend  above  and  below  the  tall  flat. 

Rectangular  tall  flat  Is  low-mounted  on  fuselage. 


frypeliy^Menu  f  indeH  ||  Shape  |[  User  |f 
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ALPHA  JET 


Country  o  f  Origin.  Franc*,  Wast  Germany. 

Similar  Aircraft.  Hawk. 

Crew.  Two 

Role.  Light  attack,  advanced  trainer. 


Armament. 


Gun  pods,  bombs,  rockets,  missiles. 


Dimensions.  Length  40  feet,  span  30  feet 

Wings.  High-mounted,  swept-back,  and  tapered  with 

curved  tips,  slight  negative  sfanL 


Flash  it  Contrast  it 


I  Sound 

Engine(s).  Two  alongside  body  under  the  wings,  oval 
shaped  air  Intakes  forward  of  the  wings'  leading  edges. 
Exhausts  at  the  rear  of  wings'  trailing  edges. 

Fusetage.  Slender,  pointed  nose  and  tail.  Two-seat 
bubble  cockpit. 


Tail.  One  Swept-back  and  tapered  tall  fin  with  squared 
tip. 

Swept-back  and  tapered  tail  flats  mid-mounted  on  body 

with  negative  slant  and  square  tips.  _ j— — 

Menu  ||  indeH  ||  Shape  ||  User  H  ^ 


J&i 


-tSSiZ 


2 


i 


CREW.  GROUP  AND  UNIT  TRAINING 


Opening  Remarks 

Aircrew  Coordination  Training  R&D 

D  r.  David  Baker,  NTSC 
Mr.  Randall  Oser,  NTSC 
Major  Wes  Woodruff,  USAF,  NTSC 
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MILITARY  RESERVES 
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LEADERSHIP 
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PERFORMANCE 
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FUTURE  RESEARCH 
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DUTY  AND  RESERVES. 


AIRCREW  COORDINATION  TRAINING 
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ORLANDO, FL 


AIRCREW  COORDINATION  TRAINING  (ACT) 

INTEGRATION 


AIRCREW  COORDINATION  TRAINING 

INTEGRATION 
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WHEN  SHOULD  AN  INTEGRATED  APPROACH 
BE  USED  OR  NOT  USED? 


AIRCREW  COORDINATION  TRAINING 

INTEGRATION 
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’EMBEDDED’  INTEGRATION 
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GROUND  I  SIM/FLT 

SCHOOL  TRAINING 
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AIRCREW  COORDINATION  TRAINING 
V-22  ACT  INTEGRATION 
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OPERATIONAL  TRAINING  ENVIRONMENT 


AIRCREW  COORDINATION  TRAINING 
V-22  ACT  INTEGRATION 


CO 

CO 

LU 


IX. 

O 


LU 

LU 

DC 


LL 

O 


CO 

§ 

< 


CO 

G 

LU 

LU 


h 

O 


o 

o 


CO 

CO 

< 


< 

o!< 

U.1 

zO 

O  o 

p  ° 

6  o 

E? 

P  ? 

z  5 

LLi  0 
Q  LU 
—  QC 


0 

CO? 

gi 

dc  cc 

r- 

LU 
CO 

O 
DC 
LU 
X 
LU 


LU 

0. 

>- 


o 

o 

I 

o 

CO 


°3 

DC 
LX  O 


DC 

O 


LU 

2 

CL 

o 


CO 


O 
LU 

cc  . 

CD  CO 


UJ  ?  DC 

lu 


LU 


Q  .2 


o  < 


i 


224 


UTILIZATION  AND  EVALUATION  OF  TRAINING 
MATERIALS 
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Once  the  ILS  mode  is  armed,  the  aircraft  will 
turn  to  the  entered  ground  track  angle  to 
intercept  the  ILS  localizer  angle.  Pilots 
should  monitor  aircraft  performance  against 
expected  aircraft  performance. 


NOTE:  1C  it  TAKEOFF  POSITION 
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AIRCREW  COORDINATION  TRAINING 

INTEGRATION 
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AIRCREW  COORDINATION  TRAINING 

INTEGRATION 
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ENVIRONMENTS 
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OVERVIEW 
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EQUIPMENT 
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SOFTWARE 


I  II  I 
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SCENARIO  DEVELOPMENT 
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AIRCREWMAN,  PAX  ON  BOARD 


SCENARIO  DEVELOPMENT  (cont) 
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FUTURE  ENHANCEMENTS/PROGRAMS 
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A-6  -  USMC  RESERVES 


TRAINING  DESIGN  AND  EVALUATION 


Instructional,  Planning  and  Evaluation  Issues 

Modeling  Skill  Acquisition: 

Dr.  Mark  Sabol 

Retention  of  Knowledge  Learned  in  College: 
Dr.  John  Ellis 


MODELS  OF  SKILL  ACQUISITION 


TOPICS  TO  BE  DISCUSSED: 
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PRELIMINARY  CONFUSION  MATRIX  RESULTS  -  SEP  90 
(Data  from  10  subjects  at  each  speed) 

All  errors  that  occurred  £5%  of  the  time  each  stimulus  was  presented 

(4-element  stimuli,  only): 


Soeed  = 

10  Groups  per  Minute 

SDeed  =  20 

GrouDS  Der  Minute 

Stimulus 

H 

•  •  •  • 

H 

•  •  •  • 

Response 

S 

•  •  • 

(38%) 

S 

•  •  •  (21%) 

Stimulus 

B 

•  •  •  • 

Response 

D 

•  •  9 

(19%) 

Stimulus 

B 

m  9  9  9 

Response 

X 

-  •  •  -  ( 6%) 

Stimulus 

V 

9  9  9  m 

V 

•  •  •  m 

Response 

U 

9  9m 

(10%) 

U 

(  7%) 

Stimulus 

F 

9  9  m  9 

Response 

L 

9  m  9  9 

(  8%) 

Stimulus 

L 

9  m  9  9 

L 

•  .  •  • 

Response 

R 

9  m  9 

(  7%) 

R 

(16%) 

Stimulus 

F 

9  9  m  9 

F 

•  •  •  • 

Response 

U 

9  9m 

(  6%) 

U 

(15%) 

Stimulus 

Y 

m  9  m  m 

Y 

-  •  -  - 

Response 

C 

m  9  m  9 

(  6%) 

C 

(5%) 

Stimulus 

C 

m  9  m  9 

Response 

Y 

-  •  -  -  (6%) 

Stimulus 

Z 

m  m  9  9 

Response 

G 

m  m  9 

(  6%) 

Stimulus 

Z 

m  m  9  9 

Response 

Q 

-  -  •  -  (  6%) 

Stimulus 

P 

9  m  m  9 

Response 

W 

(  9%) 

Stimulus 

J 

...  - 

Response 

W 

(8%) 

Stimulus 

P 

.... 

Response 

J 

•---  (7%) 
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PERCENT  CORRECT 


THREE-ELEMENT  CHARACTERS 

1 

0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 
0 


R.  .  G  _  .  D  _  .  .  S...W.__U.._K  .  O 
MORSE  CODE  CHARACTERS 


REACTION  TIME  (ms) 


THREE-ELEMENT  CHARACTERS 


SV  SIZ 


MORSE  CODE  CHARACTERS 
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PERCENT  CORRECT 


FOUR-ELEMENT  CHARACTERS 


MORSE  CODE  CHARACTERS 
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PERCENT  CORRECT 


FOUR-ELEMENT  CHARACTERS 


°17  °irl 


MORSE  CODE  CHARACTERS 


,6  £S/  Z^o  ir.  Js/s 
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FOUR-ELEMENT  CHARACTERS 


MORSE  CODE  CHARACTERS 
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Stimulation  and  Decay  of  Trace  Strength  Register 
Single  Presentation 


Rest 


Strength 


Stimulus 

Presentation 
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Stimulation  and  Decay  of  Trace  Strength  Register 
Double  Presentation 


Rest 


Stimulus 
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3-Element  Characters  with  a  Final  "dit 


Element  Characters  with  a  Final  "dah 


Character  rank  predicted 
from  trace  strength  model 


4- Element  Characters  with  a  Final  "dah 


253 


from  trace  strength  model 


4-Element  Characters  with  a  Final  "dit 
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Character  rank  predicted 
from  trace  strength  model 
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Figure  4.  Cognitive  process  model  for  skilled  Morse  code  copying. 


« 
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15  Subjects,  Last  18  GPM  Session 


Character  Duration 


PROCESSING  SEQUENCE  EXAMPLE 
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o  cd 
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CO  (0 

E  a. 

0) 


.©  0 
X5  CD 
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■o  © 
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S  f 

CD  i5 

Q.  w 
CD  C 
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C  O 
CD 

>  > 
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(D  CD 

E  a> 

CD 


co  8  £ 
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response  execution  time  (variable) 


STIMULUS  SEQUENCE  EXAMPLE 


k 
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ICi  =  Inter-Character  Interval 


CD 

3 

gi 

'll 
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Figure  1.  Diagram  of  the  Large  Start-Buffer,  Upstream  Blocking  Model  for  three  Items. 
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ASSESS  RETRAINING  DUE  TO  WAR  CONTINGENCIES 


RESEARCH  QUESTIONS 


•  How  much  do  we  remember  of  what  we  learned  in  secondary 
and  post  secondary  classroom? 


•  What  variables  affect  long-term  retention  for  knowledge 
learned  in  schools? 


•  What  cognitive  structures  and  processes  account  for  long¬ 
term  retention? 
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TYPES  OF  KNOWLEDGE  AND  SKILL 

•  Declaritive 

•  Procedural 

•  Conceptual/Contextual/Causa! 
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RESEARCH  QUESTIONS  -  KANSAS  STUDIES 


•  Will  PSI  students  learn  and  retain  more  than  LFM  students? 


•  Will  retention  for  both  groups  decline  over  time? 


•  If  PSI  do  learn  more  and  retention  does  decline,  will  the  rate 
of  decline  differ  for  the  two  groups? 


•  Will  performance  on  a  retention  test  that  is  the  same  as  the 
end-of-course  test  be  better  than  performance  on  a  different 
but  parallel  form  of  the  test? 


•  Does  the  amount  of  forgetting  differ  for  different  learning 
tasks? 


•  Does  proctoring  has  the  same  effect  as  overlearning? 
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VARIABLES  AFFECTING  RETENTION 

•  Original  Learning 

•  Task  Requirements 

•  Overlearning 

•  Test  Conditions 

•  Retention  Interval 

•  Individual  Differences 
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Data  from  PE  School  Study  -  Late  1970’s 


Test 

Version 

Immediate 

4  Weeks 
Later 

6-8  Months 
Later 

Percent 

Loss 

I 

89.28 

87.61 

73.08 

18% 

II 

87.22 

75.28 

68.93 

21% 

m 

89.64 

79.53 

70.73 

21% 

Data  from  ASW  Study  - 1983 


Condition 

Percent  Correct 

End  of  Course 

Percent  Correct 

25  Days  Later 

Percent 

Loss 

Fact 

86% 

78% 

8% 

Computation 

80% 

56% 

24% 

Gram  Analysis 

87% 

76% 

11% 

Gram  Classification 

85% 

74% 

11% 

Systematic  Analysis 

77% 

61% 

16% 

Data  from  Kansas  Study  - 1989 


Condition 

Percent  Correct 

End  of  Course 

Percent  Correct 

3  Months  Later 

Percent 

Loss 

PSI-S 

87% 

78% 

10% 

PSI-D 

73% 

16% 

LD-S 

76% 

68% 

10% 

LD-D 

62% 

17.5% 
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Item  Category  Results  from  the  Kansas  Study  - 1991 


Gains  from  Pretest  to  End  of  Course 


Item  Category  4-month  Group  11-month  Group 


Recognition 

Recall 

Comprehension 
Mental  Skills 

X 

32.1 

48.9 

30.7 

31.0 

SD 

13.6 

21.2 

20.8 

22.1 

X 

SD 

Same  Form  Loss  -  End-of-Course  to  End-of-Interval 

Item  Category 

4-month  Group 

11 -month  Group 

X 

SD 

X 

SD 

Recognition 

-13.7 

12.7 

-18.7 

11.7 

Recall 

-25.0 

20.6 

-28.3 

19.2 

Comprehension 

-12.7 

16.6 

-15.3 

17.1 

Mental  Skills 

-13.7 

18.1 

-18.7 

15.8 

Different  Form  Loss  -  End-of-Course  to  End-of-Interval 

Item  Category 

4-month  Group 

11-month  Group 

X 

SD 

X 

SD 

Recognition 

-16.9 

14.2 

-20.2 

10.8 

Recall 

-27.9 

25.4 

-36.2 

20.8 

Comprehension 

-18.6 

18.9 

-16.9 

18.8 

Mental  Skills 

-17.9 

21.0 

-21.3 

23.4 
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SUBGROUP  SESSION  III 


ADVANCED  TRAINING  TECHNOLOGY 


Basic  Job  Skills  Job  Family  Tutor: 
Dr.  Ellen  Hall 

Issues  in  Designing  and  Intelligent, 
NLP-based  Tutor  for  Foreign  Languages 
Dr.  Michelle  Sams 
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Basic  Job  Skills 
Job  Family  Tutor 
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AL/HRMJC 


History  of  BJS  Program 
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Illustration  of  JFT  Instruction 


CHRONICLE  OF  SIGNIFICANT  EVENTS 
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ying  units 


ADVOCACY  FOR  BJS  PROGRAM 


CHRONICLE  OF  SIGNIFICANT  EVENTS 

(Cont’d) 
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THE  PROBLEM 
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THE  PROBLEM 
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COMPLEX  SKILLS 
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Year*  of  Experience 
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R&D  APPROACH 


ANALYSIS 


BJS  TECHNOLOGIES 
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EVALUATION  RESULTS: 

AVIONICS  TROUBLESHOOTING  TUTOR  (SJT) 
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PROPOSED  JOB  FAMILY  TUTORS 
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Alternative  Avionics  JFT  (F15  Flightline) 


COGNITIVE  SKILL  COMMONALITIES 
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how-it-works’  ’how-to-do-it 


AVIONICS  JFT  LEARNING  STUDY 
49TH  TFW— HOLLOMAN  AFB  NM 
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-  Two  technicians  regressed 
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the  output? 


PERCENTAGE  OF  QUESTION  TYPE  BY  PROBLEM 
(POORER  LEARNERS) 
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LEARNING  PROBLEM 
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ADAPTIVE  EXPERTISE 
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In  the  course  of  working  through  RIVTEK’s  scenarios,  you 
will  be  evaluated  at  various  points  along  the  way. 
Performance  indicators  will  gauge  the  following:  (a)  your 
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INFORMATION  BOX  GENERAL  ACTION8 
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INFORMATION  BOX  GENERAL  ACTIONS 
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Coax  Sw  Dwr 
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INFORMATION  BOX  GENERAL  ACTIONS 
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S15 

Coaxial  Switch 


POST-PROBLEM  ACTIVITIES 
IN  RIVTEK 
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adaptive  expertise  presented/scored 


PAYOFF 
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Mental  Tools  for  Thinking 


PAYOFFS 
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Reduced  training  demands/costs 
Enhanced  utilization  of  personnel 


ARMY  RESEARCH  INSTITUTE  (ARI) 
FOREIGN  LANGUAGE  TUTOR 


ARI  has  a  program  of  research  on  technology  applications  for  language  learning.  A 
current  project  has  produced  an  interactive  PC-based  tutor  to  help  teach  military  specific 
language  skills  and  maintain  general  language  proficiency  for  students  at  the 
intermediate  level  and  above.  The  current  tutor  is  in  German. 


INDIVIDUALIZED  INSTRUCTION: 

The  computer  program  can  ask  questions  (written  or  orally)  and  give  feedback.  It 
can  analyze  and  understand  freely-typed  input  by  the  students  in  the  target  language 
using  natural  language  processing.  It  can  diagnose  and  track  errors  in  grammar  and 
meaning  and  adapt  the  lesson  automatically  to  the  individual  student’s  progress. 


FLEXIBILITY: 

The  tutor  includes  an  authoring  interface  which  allows  instructors  to  create  new 
lessons.  The  interface  is  also  an  aid  to  researchers  who  can  use  the  tutor  to  define 
alternative  instructional  strategies  and  test  them.  This  interface  is  comprised  of  sample 
templates  to  be  filled  in  and  requires  no  programming  skills. 


EXTENDABILITY: 

The  tutors  under  development  are  designed  to  challenge  the  student  to  communicate 
and  learn  in  the  target  language.  Most  of  the  software  in  these  tutors  would  require  only 
minimal  changes  to  apply  to  a  range  of  foreign  language  (extendability  to  Arabic  has 
been  demonstrated).  This  is  primarily  due  to  two  factors:  1)  the  natural  language 
processor  contains  reusable  language  universal  components,  and  2)  the  lesson  authoring 
system  (a  "one-time"  development  cost)  can  be  used  for  any  language  application. 

FUTURE  DEVELOPMENT: 

A  new  3-year  development  program  will  begin  in  1992,  with  the  goal  of  creating  a 
"second  generation"  tutor  that  will  immerse  the  student  in  the  language  environment 
through  interactive  dialogs  utilizing  natural  language  processing,  speech  recognition 
systems,  and  dynamic  graphics.  Spanish  and  Arabic  are  the  target  languages. 


ARI  POCs:  Dr.  Melissa  Holland,  Team  Leader 

Dr.  Jonathan  Kaplan,  Dr.  Michelle  Sams,  Dr.  Cathie  Alderks,  Mr.  Rich  Maisano 
(703)  274-5540  AV  284-5540 
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RESEARCH  INSTITUTE 
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BACKGROUND 
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language  acquisition  and  retention 


ADVANCED  TECHNOLOGY  FOR  LANGUAGE  LEARNING 
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(Spanish  &  Arabic,  3  year  program  starts  May  1992) 


LANGUAGE  SKILLS 
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|  listen  ->  pronounce 

Goal  is  to  communicate. 


OUTSTANDING  FEATURES  OF  TUTOR 
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Parser  is  based  on  language  universal  principles  with  "switches' 
that  change  the  parameters  for  different  languages 


SECOND  GENERATION  ATTRIBUTES 
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purposes  and  lesson  alteration 


RESEARCH:  POSSIBLE  ISSUES  TO  BE  EXAMINED 
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Speech  recognition/production 


CREW.  GROUP  AND  UNIT  TRAINING 


Update  of  AF  ISD  Process: 
Major  Conrad  Bills 


Team  Decision-Making  Training  (Update) 
Dr.  Eduardo  Salas 
(No  hard  copies  available) 
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INTRODUCTION 
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REVISION  OF  AIR  FORCE  ISD 
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REVISION  OF  AIR  FORCE  ISD 
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REVISION  OF  AIR  FORCE  ISO 
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REVISION  OF  AIR  FORCE  ISO 
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REVISION  OF  AIR  FORCE  ISD 
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REVISION  OF  AIR  FORCE  ISD 


RECOMMENDATIONS 
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RECOMMENDATIONS 
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REVISION  OF  AIR  FORCE  ISD 


RECOMMENDATIONS 
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RECOMMENDATIONS 
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RECOMMENDATIONS 
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NEXT-PHASE  ACTIVITIES 
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REVISION  OF  AIR  FORCE  ISD 
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REVISION  OF  AIR  FORCE  ISD 


EVENT  JAN  FEB  MAR  APR  MAY  JUN 
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Draft  AFM  50-2 


G  SEP  OCT 
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Executive  Summary 


Camera-Ready  Copy 
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LEGEND 

TRIPS: 

1  1  SwRI  Work  Effort 

i  Wright-PattBrson  (ASD) 

2.  LukaAFB 

U/Z/X  Conduct  Ponwadre  Evtuationt 
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Training  Systea  Functional  Nodal 
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INING  DESIGN  AND  EVALUATION 


Instructional,  Planning  and  Evaluation  Issues 

Instructional  Strategies  for  Logistic 
Command  and  Control: 

Captain  Reynold  Hioki 

Distance  Learning: 

Mr.  Dennis  Gettman 
(No  hard  copies  available) 
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Training  Technology  Technical 
Group  (T2TG) 

Desktop  Training  for 
Logistics  Command  and 
Control 


Capt  Reynold  Hioki 
AL/HRTC 
25  Mar  92 


PROBLEM  STATEMENT 

-  Limited  training  opportunities  for  required  complex, 
ti  me/risk-critical  decision-making. 

-  -  Expense  and  other  limitations  of  exercises 

•  -  Changing  threat  environment 

-  -  Personnel  changeover 

•  -  Use  of  battle  staff  augmentees 

•  -  Inadequate  knowledge  about  effective  instructional 

strategies  for  complex  decision  making 

-  •  Lack  of  validated  training  outcomes 


-  Sponsor:  HQ  USAF/LGXX 

-  User:  HQ  AFLC 
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SYSTEM  ARCHITECTURE 


PAYOFFS 


Provide  effective  complex  decision-making 
training  for  individual  battle  staff  personnel: 


when  needed 
where  needed 
at  lower  cost 


PRODUCTS 


Validated  instructional  strategies 

--  literature-based 

-  contractor  sponsored  symposium 

-  experimentation 

ITS  prototype 

-  object-based,  graphical  user  interface 

-  adaptive  to  individual  student  needs 

-  presentation  and  practice  capability 


SCIENTIFIC  METHODOLOGY 

Instructional  methodology 

-  training  requirements  analysis 

-  literature  review 

-  instructional  methodology  symposium 

-  field  evaluation 

-  experimentation 

ITS  prototype 

-  instructional  methodology 

-  rapid-prototyping  approach 

-  object-based 

-  field  evaluation 

-  experimentation 


WRAP-UP 


Training  Design  and  Evaluation 
Summary  -  Mark  Teachout 
(No  hard  copies  available) 

CREW.  GROUP.  UNIT  AND  TEAM 

Dr.  Eduardo  Salas 

ADVANCED  TRAINING  TECHNOLOGY 
Dr .  Ray  Perez 
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T2TG 

Crew,  Group,  Team,  &  Unit  Training  Technology 


Eduardo  Salas  &  Frank  Moses 
Co-Chairs 


-  Session  I  was  entirely  devoted  to  a  discussion  of  a  DMSO 
proposal  submitted  by  ARI  (lead) ,  NTSC,  and  Armstrong  Lab.  The 
discussion  was  led  by  Frank  Moses.  He  outlined  the  objectives  of 
the  project  and  the  products.  The  proposed  work  is  to  exploit 
"SIMNET-like"  technology  and  demonstrated  the  efficacy  of 
different  training  strategies.  He  also  showed  a  videotape 
recreating  the  Battle  of  73  Easting. 

-  The  topic  for  Session  II  was  aircrew  coordination  training 
(ACT) .  There  was  a  presentation  given  by  Major  Woodruff  on  a 
Tabletop  Aircrew  Coordination  trainer.  This  is  a  low-fidelity  PC 
based  flight  simulator  that  allows  crews  to  practice  teamwork 
skills.  It  generated  lots  of  interest  and  discussion.  Then, 
David  Baker  and  Randy  Oser  from  NTSC  updated  the  group  on  recent 
advances  of  the  ACT  research.  They  focused  their  discussion  on 
organizational  issues  in  ACT  and  integration  of  ACT  to  technical 
skills.  Finally,  Judith  Orasanu  (NASA-Ames)  gave  a  summary  of 
current  work  that  they  are  supporting  on  aircrew  coordination  and 
presented  some  data  on  her  research. 

-  The  last  session  was  a  presentation  by  Major  Bills  on  the  AF 
ISD  work.  He  updated  the  group  on  what  has  been  done  and  where 
the  work  is  going. 
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Summary  Advanced  Training  Technology  Subgroup  of  the  T2T6 


The  Advanced  Training  Technology  subgroup  met  on  the 
24  &  25  March  1992.  The  thirteen  attendees  for  this  session 
represented  both  bench  scientists  and  users.  They  represented 
all  the  major  laboratories  in  the  three  services  (NRL,  ARI , 

AFHRL,  NTSC,  NPDRC,  ONR) .  The  theme  of  the  five  paper 
presentations  of  this  meeting  was  research  on  "Virtual  Reality 
Its  Application  to  Training  and  Intelligent  Tutors."  Five  papers 
were  presented  by  Scientists  from  the  service  laboratories  (ARI, 
AFHRL,  NTSC,  NPDRC) .  The  guest  discussant  for  these 
presentations  was  Dr.  Denis  Breglia  from  NTSC.  Each  presentation 
was  followed  by  a  discussion  led  by  Dr.  Perez,  Ms.  Dickieson,  and 
Dr.  Breglia.  The  meeting  was  co-chaired  by  Dr.  Perez  and  Ms. 
Dickieson,  Ms.  Dickieson  is  the  incoming  chair. 

Dr.  Richard  Thurman  (AFHRL)  presented  his  research  on  the 
use  of  Virtual  Reality  (VR)  technology  to  enhance  pilot  tactical 
skills.  He  was  followed  by  a  presentation  by  Dr.  Bruce  Knerr 
(ARI)  on  the  Army's  efforts  to  generate  training  requirements  for 
the  use  of  Virtual  Reality  technology  in  the  Army's  future  Close 
Combat  Tactical  Trainer.  Dr.  Joseph  Psotka  (ARI)  presented  a 
paper  on  the  use  of  hyper-media  to  enhance  visual  problem-solving 
in  a  Virtual  Reality  environment.  Dr.  Psotka  pointed  out  the 
similarities  in  considerations  for  VR  and  Hypertext  applications. 
He  was  followed  by  a  paper  presented  by  Dr.  Ellen  Hall  (AFHRL)  on 
the  Air  Force's  tutor  for  the  family  of  skills.  This  paper  in 
turn  was  followed  by  Dr.  Wisher  (ARI)  who  described  his  research 
on  cognitive  modeling  of  the  acquisition  of  morse  code.  Dr. 
Michelle  Sams  (ARI)  presented  the  work  they  are  doing  on  a 
Foreign  Language  Tutor.  The  objective  of  Dr.  Hall  and  Dr.  Sams' 
projects  is  to  design,  develop,  and  implement  intelligent  tutors. 

The  paper  presentations  and  questions  were  followed  by  a 
discussion  and  suggestions  of  future  topics  for  Advanced  Training 
Technologies  Subgroup  meetings. 

A  summary  of  conclusions  of  the  VR  research  and  comments 
made  by  the  group  follow. 

o  Little  or  no  instructional  design  theory  exists  to  go 
along  with  and  guide  the  use  of  VR  technology. 

o  Hardware/software  hasn't  reach  sufficient  level  of 
maturity  for  commercial  or  military  applications. 

Commercial  applications  will  produce  the  largest 
technical  gains  in  the  development  of  VR. 

No  economies  of  scales  exists  for  VR. 

o  More  research  is  needed  to  answer  the  following  questions 
and  issues. 
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To  what  degree  do  we  need  this  technology  (what  current 
training  problem  will  it  solve)? 

We  need  a  better  understanding  of  spatial  orientation 
before  we  can  effectively  design  and  use  VR  technology. 

We  need  better  theories  and  definitions  of  visual 
metaphors  for  capturing  data. 

o  Government  needs  to  be  more  proactive  in  helping  industry 
define  its  needs. 

In  general,  advances  in  training  technology,  including  the 
work  on  the  design  of  Intelligent  tutors,  need  to  address  the 
requirement  for  personnel  to  perform  increasingly  complex  and 
difficult  tasks  and  possess  a  wide  range  of  skills. 

Topics  for  next  meeting: 

o  Distance  Learning/Distance  Education 

o  Virtual  Environments 

o  Applied  Technologies 

It  was  suggested  that  we  invite  DMSO  to  attend  and  have 
personnel  from  Disney  World  brief  on  their  applications  of  VR. 
Everyone  agreed  that  we  should  always  have  included  in  our 
sessions,  various  projects  to  be  briefed  that  are  beyond  the 
conceptual  stage  and  can  report  data. 

In  sum,  the  meeting  was  very  successful  as  measured  by  the 
enthusiasms  of  the  participants  and  their  representativeness.  At 
least  one  bench  scientist  from  each  of  the  services  labs  was  in 
attendance . 


Ray  S.  Perez 
Jan  Dickieson 
Co-Chairpersons  Advanced 
Training  Technologies  Subgroup 
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ADVANCE  TRAINING  TECHNOLOGIES  ATTENDEES 


NAME 

AGENCY/SYMBOL 

PHONE 

Ray  Perez 

ARI,  PERI-IIB 

C  (703)  274-8694 
DSN  284-8694 

Jan  Dickieson 

NPRDC ,  132 

C  (619)  553-9270 
AV  553-9270 

Mijor  Jim  Mika 

TPDC 

C  (407)  281-3683 

John  E.  Buckley 

TRADOC-DCST 

AV  680-5535 

Was  Regian 

AL/HRTI 

AV  240-2034 

Jim  Fleming 

AL/HRTI 

AV  240-2034 

Richard  Thurman 

AL/HRAU 

DSN  474-6561 

Philip  Miller 

436STS/OLED  Edwards  AFB 

DSN  525-8767 

Kevin  Dixon 

HQATC/XPCR 

DSN  487-3390 

Michelle  Sams 

ARI,  Alex  VA 

AV  284-5540 

Reynold  Hiola 

AL/HRTC 

DSN  240-2981 

i-F'ryl  Baker 

NPRDC,  01G 

C  (619)  553-0785 
AV  553-0785 

Joe  Psotka 

ARI,  Alex  VA  C  (703)  274-5540 

AV  284-5540 

Psotka@ALEXANDRIA-EMH2 . ARMY . ML 

Bruce  Knerr 

ARI-PM  TRADE  Field  Unit 

C  (407)  380-4378 
AV  960-4378 

D'-nis  Breglia 

NTSC-CODE  251 

C  (407)  380-8159 

AV  960-8159 

C  i  r ]  Dr i skel 1 

US  Army  STRICOM 
(Formerly  PM  TRADE) 
AMCPM-TND-E 

C  (407)  380-4348 

DSN  960-4348 
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T2TG  REGISTRATION  LIST 


ATTENDEES 


BAKER,  DAVID  DR. 

BAKER,  MERYL  DR. 

BILLS,  CONRAD  G.  MAJ 
BREGLIA,  DENIS  MR. 
BUCKLEY,  JOHN  MR. 
CARROLL,  LYNN  LTC 
CHAMBERLAIN,  MARTY  CAPT 
CRAWFORD,  ALICE  MS. 
DICKIESON ,  JANET  MS. 
DIXON,  KEVIN  CAPT 
DRILLINGS,  MICHAEL  DR. 
DRISKELL,  CARL  MR. 
ELLIS,  JOHN  DR. 

FLEMING,  JIM  MR. 
GETTMAN,  DENNIS  MR. 
GRAY,  THOMAS  DR. 

HALL,  ELLEN  DR. 

HIOKI ,  REYNOLD  CAPT 
HOWELL,  WILLIAM  DR. 
KNERR,  BRUCE  DR. 

MIKA,  JAMES  MAJ 
MILLER,  PHILIP  CAPT 
MITCHELL,  DENNIS  LTC 
MORALES,  MICHELE  MS. 
MOSES,  FRANK  DR. 
ORASANU,  JUDITH 
OSER,  RANDALL  MR. 
THURMAN,  RICHARD  DR. 
PEREZ,  RAY  DR. 

PSOTKA,  JOE  DR. 

REG I AN,  WES  DR. 

ROMAN I CK,  PAUL  D.  MAJ 
RUCK,  HENDRICK  DR. 
SABOL,  MARK  DR. 

SALAS,  EDUARDO  DR. 

SAMS,  MICHELLE  DR. 
SEIDEL,  ROBERT  J.  DR. 
SINE,  DON  CDR 
TEACHOUT,  MARK  DR. 
UTCHEL,  DONALD  CAPT 
WOODRUFF,  WES  MAJ 


NTSC,  Orlando,  FL 

NPR&DC,  San  Diego,  CA 

ASD/YTEE,  Wright-Patterson  AFB,  OH 

NTSC,  CODE  251,  Orlando,  FL 

HQTRADOC,  Ft.  Monroe,  VA 

ATRD,  Williams  AFB,  AZ 

ONR,  Arlington,  VA 

NPS ,  Monterey,  CA 

NPRDC,  San  Diego,  CA 

ATC/XPCR,  Randolph  AFB,  TX 

ARI,  Alexandria,  VA 

PM-TRADE,  Orlando,  FL 

NPRDC,  San  Diego,  CA 

AL/HRTI ,  Brooks  AFB,  TX 

AL/HRTC,  Brooks  AFB,  TX 

AL/HRA,  Williams  AFB,  AZ 

AL/HRMJC,  Brooks  AFB,  TX 

AL/HRTC,  Brooks  AFB,  TX 

HR/CA,  Brooks  AFB,  TX 

PM-TRADE  ARI,  Orlando,  FL 

TPDC,  Orlando,  FL 

436  STS/OLED,  Edwards,  CA 

ASD/XRS ,  Wright-Patterson  AFB,  OH 

AL/HRTE ,  Brooks  AFB,  TX 

ARI,  Alexandria,  VA 

NASA-ARC,  Moffett  Field,  CA 

NTSC,  Orlando,  FL 

AL/HRAU,  Williams  AFB,  AZ 

ARI,  Alexandria,  VA 

ARI,  Alexandria,  VA 

AL/HRTI,  San  Antonio,  TX 

436  STS/OL-ED,  Edwards  AFB,  CA 

AL/HRT,  Brooks  AFB,  TX 

ARI,  Alexandria,  VA 

NTSC,  Orlando,  FL 

ARI,  Alexandria,  VA 

ARI,  Alexandria,  VA 

ATC,  Dahlgren,  VA 

AL/HRTE,  San  Antonio,  TX 

436  STS/OL-ED  Edwards  AFB,  CA 

NTSC,  Orlando,  FL 


356 


